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Abstract 

This paper introduces a simple and very general theory of compressive sensing. In this 
theory, the sensing mechanism simply selects sensing vectors independently at random from a 
probability distribution F; it includes all models — e.g. Gaussian, frequency measurements — 
discussed in the literature, but also provides a framework for new measurement strategies as 
well. We prove that if the probability distribution F obeys a simple incoherence property and 
an isotropy property, one can faithfully recover approximately sparse signals from a minimal 
number of noisy measurements. The novelty is that our recovery results do not require the 
restricted isomctry property (RIP) — they make use of a much weaker notion — or a random 
model for the signal. As an example, the paper shows that a signal with s nonzero entries can 
be faithfully recovered from about slogn Fourier coefficients that are contaminated with noise. 

Keywords. Compressed sensing, l\ minimization, the LASSO, the Dantzig selector, (weak) 
restricted isometries, random matrices, sparse regression, operator Bernstein inequalities, Gross' 
golfing scheme. 

Dedicated to the memory of Jerrold E. Marsden. 

1 Introduction 

This paper develops a novel, simple and general theory of compressive sensing [12, 15, 19], a rapidly 
growing field of research that has developed protocols for acquiring certain types of signals with 
far fewer data bits than what is classically accepted. 

1.1 A RIPless theory? 

The early paper [12] triggered a massive amount of research by showing that it is possible to 
sample signals at a rate proportional to their information content rather than their bandwidth. For 
instance, in a discrete setting, this theory asserts that a digital signal x € [R n (which can be viewed as 
Nyquist samples of a continuous-time signal over a time window of interest) can be recovered from 
a small random sample of its Fourier coefficients provided that x is sufficiently sparse. Formally, 
suppose that our signal x has at most s nonzero amplitudes at completely unknown locations - 
such a signal is called s-sparse — and that we are given the value of its discrete Fourier transform 
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(DFT) at m frequencies selected uniformly at random (we think of m as being much smaller than 
re). Then [12] showed that one can recover x by solving an optimization problem which simply finds, 
among all candidate signals, that with the minimum l\ norm; the number of samples we need must 
be on the order of slogn. In other words, if we think of s as a measure of the information content, 
we can sample nonadaptively nearly at the information rate without information loss. By swapping 
time and frequency, this also says that signals occupying a very large bandwidth but with a sparse 
spectrum can be sampled (at random time locations) at a rate far below the Shannon-Nyquist rate. 

Despite considerable progress in the field, some important questions are still open. We discuss 
two that have both a theoretical and practical appeal. 

Is it possible to faithfully recover a nearly sparse signal one which is well ap- 

proximated by its s largest entries, from about s log re of its Fourier coefficients ? Is it 
still possible when these coefficients are further corrupted by noise? 

These issues are paramount since in real-world applications, signals are never exactly sparse, and 
measurements are never perfect either. Now the traditional way of addressing these types of 
problems in the field is by means of the restricted isometry property (RIP) [14]. The trouble here 
is that it is unknown whether or not this property holds when the sample size m is on the order 
of slogn. In fact, answering this one way or the other is generally regarded as extremely difficult, 
and so the restricted isometry machinery does not directly apply in this setting. 

This paper proves that the two questions formulated above have positive answers. In fact, we 
introduce recovery results which are — up to a logarithmic factor — as good as those one would 
get if the restricted isometry property were known to be true. To fix ideas, suppose we observe m 
noisy discrete Fourier coefficients about an s-sparse signal x, 

n-l 

y k =Y i e* l * Ukt x[t] + az k , fc = l,...,m. (1.1) 
t=o 

Here, the frequencies Uk are chosen uniformly at random in {0, 1/ra, 2/n, . . . , (n - l)/n} and Zk is 
white noise with unit variance. Then if the number of samples m is on the order of slog re, it is 
possible to get an estimate x obeying 

\\x - x\1 = polylog(n) —a 2 (1.2) 
m 

by solving a convex ^-minimization program. (Note that when the noise vanishes, the recovery is 
exact.) Up to the logarithmic factor, which may sometimes be on the order of logn and at most 
a small power of this quantity, this is optimal. Now if the RIP held, one would get a squared 
error bounded by 0(logre)^<r 2 [16, 5] and, therefore, the 'RIPless' theory developed in this paper 
roughly enjoys the same performance guarantees. 

1.2 A general theory 

The estimate we have just seen is not isolated and the real purpose of this paper is to develop a 
theory of compressive sensing which is both as simple and as general as possible. 

At the heart of compressive sensing is the idea that randomness can be used as an effective 
sensing mechanism. We note that random measurements are not only crucial in the derivation of 
many theoretical results, but also generally seem to give better empirical results as well. Therefore, 
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we propose a mechanism whereby sensing vectors are independently sampled from a population F. 
Mathematically, we observe 

y k = (a k ,x) + az k , fc = l,...,m, (1.3) 

where x e R n , {z k } is a noise sequence, and the sensing vectors a k ~ F. For example, if F is the 
family of complex sinusoids, this is the Fourier sampling model introduced earlier. All we require 
from F is an isotropy property and an incoherence property. 

Isotropy property: We say that F obeys the isotropy property if 

Eaa* = I, a~F. (1.4) 

If F has mean zero (we do not require this), then Eaa* is the covariance matrix of F. In 
other words, the isotropy condition states that the components of a ~ F have unit variance 
and are uncorrelated. This assumption may be weakened a little, as we shall see later. 

Incoherence property: We may take the coherence parameter /u(-F) to be the smallest number 
such that with a = (a[l], . . . , a[^]) ~ F, 

m^\a[tf<^F) (1.5) 

l<t<n 

holds either deterministically or stochastically in the sense discussed below. The smaller 
/i(F), i.e. the more incoherent the sensing vectors, the fewer samples we need for accurate 
recovery. When a simple deterministic bound is not available, one can take the smallest scalar 
(j, obeying 

EfrT 1 \\af l2 l E c] < ^rT 3/2 and P(E C ) < (nm)" 1 , (1.6) 
where E is the event {maxi<t< n |a[i]| 2 > fi}. 

Suppose for instance that the components are i.i.d. A/"(0, 1). Then a simple calculation we shall 
not detail shows that 

Ein- 1 \\a\\l t E c] <2nP(Z>^7) + 2^(v7I), (1.7) 
P(E C ) < 2nP(Z> ^//7), 

where Z is standard normal and <j> is its density function. The inequality P{Z > t) < cp(t)/t shows 
that one can take fJ-(F) < 61ogn as long as n > 16 and m < n. More generally, if the components 
of a are i.i.d. samples from a sub-Gaussian distribution, fi(F) is at most a constant times log??,. If 
they are i.i.d. from a sub-exponential distribution, n(F) is at most a constant times log 2 n. In what 
follows, however, it might be convenient for the reader to assume that the deterministic bound (1.5) 
holds. 

It follows from the isotropy property that E|a[t]| = 1, and thus fJ*(F) > 1. This lower bound is 
achievable by several distributions and one such example is obtained by sampling a row from the 
DFT matrix as before, so that 

a[t] = e M/n , 

where k is chosen uniformly at random in{0,l,...,ra-l}. Then another simple calculation shows 
that Eaa* = /and fi(F) = 1 since |a[i]| 2 = 1 for all t. At the other extreme, suppose the measurement 
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process reveals one entry of x selected uniformly at random so that a = sfne-i where i is uniform 
in {1, . . . , n}; the normalization ensures that Eaa* = /. This is a lousy acquisition protocol because 
one would need to sample on the order of nlogra times to recover even a 1-sparse vector (the 
logarithmic term comes from the coupon collector effect). Not surprisingly, this distribution is in 
fact highly coherent as n(F) = n. 

With the assumptions set, we now give a representative result of this paper: suppose x is an 
arbitrary but fixed s-sparse vector and that one collects information about this signal by means of 
the random sensing mechanism (1.3), where z is white noise. Then if the number of samples is on 
the order fi(F)s\ogn, one can invoke t\ minimization to get an estimator x, obeying 

lis - acllf < polylog(n) —a 2 . 

m 

This bound is sharp. It is not possible to substantially reduce the number of measurements and 
get a similar bound, no matter how intractable the recovery method might be. Further, with this 
many measurements, the upper bound is optimal up to logarithmic factors. Finally, we will see 
that when the signal is not exactly sparse, we just need to add an approximation error to the upper 
bound. 

To summarize, this paper proves that one can faithfully recover approximately s-sparse signals 
from about slogn random incoherent measurements for which /(x(-F) = O(l). 

1.3 Examples of incoherent measurements 

We have seen through examples that sensing vectors with low coherence are global or spread out. 
Incoherence alone, however, is not a sufficient condition: if F were a constant distribution (sampling 
from F would always return the same vector), one would not learn anything new about the signal by 
taking more samples regardless of the level of incoherence. However, as we will see, the incoherence 
and isotropy properties together guarantee that sparse vectors lie away from the nullspace of the 
sensing matrix whose rows are the a*,'s. 

The role of the isotropy condition is to keep the measurement matrix from being rank defi- 
cient when sufficiently many measurements are taken (and similarly for subsets of columns of A). 
Specifically, one would hope to be able to recover any signal from an arbitrarily large number of 
measurements. However, if Eaa* were rank deficient, there would be signals i e P that would 
not be recoverable from an arbitrary number of samples; just take x t in the nullspace of Eaa*. 
The nonnegative random variable x*aa*x has vanishing expectation, which implies a*x - almost 
surely. (Put differently, all of the measurements would be zero almost surely.) In contrast, the 
isotropy condition implies that — YJk=X a k a k ~* I almost surely as m -*■ oo and, therefore, with 
enough measurements, the sensing matrix is well conditioned and has a left-inverse. 1 

We now provide examples of incoherent and isotropic measurements. 

• Sensing vectors with independent components. Suppose the components of a ~ F 
are independently distributed with mean zero and unit variance. Then F is isotropic. In 
addition, if the distribution of each component is light-tailed, then the measurements are 
clearly incoherent. 

One could require 'near isotropy,' i.e. Eaa* ~ I. If the approximation were tight enough, our theoretical results 
would still follow with minimal changes to the proof. 
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A special case concerns the case where a ~ 7V(0,I), also known in the field as the Gaussian 
measurement ensemble, which is perhaps the most commonly studied. Here, one can take 
fj.(F) = 61ogn as seen before. 

Another special case is the binary measurement ensemble where the entries of a are symmetric 
Bernoulli variables taking on the values ±1. A shifted version of this distribution is the sensing 
mechanism underlying the single pixel camera [22] . 

• Subsampled orthogonal transforms: Suppose we have an orthogonal matrix obeying 
U*U = nl. Then consider the sampling mechanism picking rows of U uniformly and inde- 
pendently at random. In the case where U is the DFT, this is the random frequency model 
introduced earlier. Clearly, this distribution is isotropic and /J>(F) - maxjj |C/jj| 2 . In the case 
where U is a Hadamard matrix, or a complex Fourier matrix, fJ,(F) = 1. 

• Random convolutions: Consider the circular convolution model y = Gx in which 



G = 



g[0] g[l] g[2] ... g[n-l] 
g[n-l] g[0] <?[!] 



L <?[!] ••• g[n-l] g[0] J 



Because a convolution is diagonal in the Fourier domain (we just multiply the Fourier compo- 
nents of x with those of g), G is an isometry if the Fourier components of g = (g[0], . . . , g[n-l]) 
have the same magnitude. In this case, sampling a convolution product at randomly se- 
lected time locations is an isotropic and incoherent process provided g is spread out (fJ-(F) = 
maxf \g(t)\ ). This example extends to higher dimensions; e.g. to spatial 3D convolutions. 

• Subsampled tight or continuous frames: We can generalize the example above by sub- 
sampling a tight frame or even a continuous frame. An important example might be the 
Fourier transform with a continuous frequency spectrum. Here, 

a(t) = e , 

where u is chosen uniformly at random in [0, 1] (instead of being on an equispaced lattice 
as before). This distribution is isotropic and obeys n(F) = 1. A situation where this arises 
is in magnetic resonance imaging (MRI) as frequency samples rarely fall on an equispaced 
Nyquist grid. By swapping time and frequency, this is equivalent to sampling a nearly sparse 
trigonometric polynomial at randomly selected time points in the unit interval [32]. 

These examples could of course be multiplied, and we hope we have made clear that our framework 
is general and encompasses all the measurement models commonly discussed in compressive sensing 
— and perhaps many more. 

1.4 Matrix notation 

Before continuing, we pause to introduce some useful matrix notation. Divide both sides of (1.3) 
by \fm, and rewrite our statistical model as 

y = Ax + a m z; (1.8) 
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the kth entry of y is yu divided by \pm, the fcth row of A is a* k divided by \/m, and a m is a divided 
by \fm. This normalization implies that the columns of A are approximately unit-normed, and is 
most used in the compressive sensing literature. 

1.5 Incoherent sampling theorem 

For pedagogical reasons, we introduce our results by first presenting a recovery result from noiseless 
data. The recovered signal is obtained by the standard ^-minimization program 



(Recall that the rows of A are normalized independent samples from F.) 

Theorem 1.1 (Noiseless incoherent sampling) Letx be a fixed but otherwise arbitrary s- sparse 
vector in R n . Then with probability at least 1 - 5/n - e~@ , x is the unique minimizer to (1.9) with 
y = Ax provided that 

m> Cp ■ (j,(F) ■ s ■ log n. 
More precisely, Cp may be chosen as Cq(1 + (3) for some positive numerical constant Cq. 

Among other things, this theorem states that one can perfectly recover an arbitrary sparse signal 
from about slogn convolution samples, or a signal that happens to be sparse in the wavelet domain 
from about slogn randomly selected noiselet coefficients. It extends an earlier result [11], which 
assumed a subsampled orthogonal model, and strengthens it since that reference could only prove 
the claim for randomly signed vectors x. Here, x is arbitrary, and we do not make any distributional 
assumption about its support or its sign pattern. 

This theorem is also about a fundamental information theoretic limit: the number of samples 
for perfect recovery has to be on the order of fi(F) ■ s-logn, and cannot possibly be much below this 
number. More precisely, suppose we are given a distribution F with coherence parameter ^{F). 
Then there exist s-sparse vectors that cannot be recovered with probability at least 1 - 1/n, say, 
from fewer than a constant times /(x(-F) ■ s ■ logn samples. When /(x(-F) = 1, this has been already 
established since [12] proves that some s sparse signals cannot be recovered from fewer than a 
constant times s ■ logn random DFT samples. Our general claim follows from a modification of 
the argument in [12]. Assume, without loss of generality, that fi(F) is an integer and consider the 
isotropic process that samples rows from annxn block diagonal matrix, each block being a DFT of 
a smaller size; that is, of size n/£ where ^{F) = t. Then if m < cq ■ fJ,(F) ■ s ■ logn, one can construct 
s-sparse signals just as in [12] for which Ax = with probability at least 1/n. We omit the details. 

The important aspect, here, is the role played by the coherence parameter i-i(F). In general, the 
minimal number of samples must be on the order of the coherence times the sparsity level s times a 
logarithmic factor. Put differently, the coherence completely determines the minimal sampling rate. 

1.6 Main results 

We assume for simplicity that we are undersampling so that m < n. Our general result deals with 
1) arbitrary signals which are not necessarily sparse (images are never exactly sparse even in a 
transformed domain) and 2) noise. To recover x from the data y and the model (1.8), we consider 
the unconstrained LASSO which solves the l\ regularized least-squares problem 



min llxIL subject to Ax-y. 




mm 



Ax-y\\ i2 + Xa m \\x 



(1.10) 
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We assume that z is Gaussian z ~ N(0,I). However, the theorem below remains valid as long as 
11^4*^11^ < A n for some A n > 0, and thus many other noise models would work as well. In what 
follows, x s is the best s-sparse approximation of x or, equivalently, the vector consisting of the s 
largest entries of x in magnitude. 

Theorem 1.2 Let x be an arbitrary fixed vector in R n . Then with probability at least 1 — 6/n- 6e~^ 
the solution to (1.10) with A = 10\/Iogn obeys 



\x-x\f <minC(l + a) 
1 U2 i<s<s y ' 



+ a 



slogn 



■/?? 



(1.11) 



provided that m>Cp- u(-F) ■ s ■ logn. If one measures the error in the l\ norm, then 

\x-x\n <minC(l + a) 

" Htl 1<S<S 



|X + so~ 



logn 



m 



(1.12) 



Above, C is a numerical constant, Cp can be chosen as before, and a = \/ ^ 1+/3 ^ log n which is never 
greater than log 2 n in this setup. 

These robust error bounds do not require either (1) a random model on the signal or (2) the RIP 
nor one of a few closely related strong conditions such as the RIP-1 [24], the restricted eigenvalue 
assumption [5] or the compatibility condition [38] . The conditions are weak enough that they do not 
necessarily imply uniform sparse-signal recovery, but instead they imply recovery of an arbitrary 
fixed sparse signal with high probability. Further, the error bound is within at most a log 2 n factor 
of what has been established using the RIP since a variation on the arguments in [16] would give 
an error bound proportional to the quantity inside the square brackets in (1.11). As a consequence, 
the error bound is within a polylogarithmic factor of what is achievable with the help of an oracle 
that would reveal the locations of the significant coordinates of the unknown signal [16] . In other 
words, it cannot be substantially improved. 

Because much of the compressive sensing literature works with restricted isometry conditions - 
we shall discuss exceptions such as [21, 4] in Section 1.7 - we pause here to discuss these conditions 
and to compare them to our own. We say that an m x n matrix A obeys the RIP with parameters 
s and 5 if 



(1 



II r. 



<\\Av\\l<(l + 5)\\v\\ 



(1.13) 



for all s-sparse vectors v. In other words, all the submatrices of A with at most s columns are well 
conditioned. When the RIP holds with parameters 2s and 5 < 0.414 ... [8] or even 5 < 0.453 . . . [23], 
it is known that the error bound (1.11) holds (without the factor (1 + a)). This 5 is sometimes 
referred to as the restricted isometry constant. 

Bounds on the restricted isometry constant have been established in [15] and in [35] for partial 
DFT matrices, and by extension, for partial subsampled orthogonal transforms. For instance, [35] 
proves that if A is a properly normalized partial DFT matrix, then the RIP with 5 = 1/4 holds with 
high probability if m > C ■ s log 4 n (C is some positive constant). We believe the proof extends with 
hardly any change to show that the measurement ensembles considered in this paper obey the RIP 
with high probability when m > C ■ /-i(F) ■ slog 4 n. Thus, our result bridges the gap between the 
region where the RIP holds and the region in which one has the minimum number of measurements 
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needed to prove perfect recovery of exactly sparse signals from noisy data, which is on the order of 
/J.(F) ■ slogn. 

The careful reader will no doubt remark that for very specific models such as the Gaussian 
measurement ensemble, it is known that on the order slogn/ s samples are sufficient for stable 
recovery while our result asserts that on the order of s log 2 n are sufficient (and s log n for the 
binary measurement ensemble). This slight loss is a small price to pay for a very simple general 
theory, which accommodates a wide array of sensing strategies. Having said this, the reader will 
also verify that specializing our proofs below gives an optimal result for the Gaussian ensemble; 
i.e. establishes a near-optimal error bound from about s log n/s observations. 

Finally, another frequently discussed algorithm for sparse regression is the Dantzig selector [16]. 
Here, the estimator is given by the solution to the linear program 



mm \\x\\ 

xeR" 



subject to |^*(Ac-y)L <Xa n 



(1.14) 



We show that the Dantzig selector obeys nearly the same error bound. 



Theorem 1.3 The Dantzig selector, with A = lOvlogn and everything else the same as in Theorem 
1.2, obeys 



\x - x\i < minC(l + a 2 ) 



x\\« < min C( 1 + a 2 ) 



/ s log n 

+ 0" 

m 



I X Xej||£^ ~\~ S(J 



log re 



m 



(1.15) 
(1.16) 



with the same probabilities as before. 

The only difference is a 2 instead of a in the right-hand sides. 



1.7 Our contribution 

The main contribution of this paper is to provide a simple framework which applies to all the 
standard compressive sensing models and some new ones as well. The results in this paper also 
reduce the minimal number of measurements theoretically required in some standard sensing models 
such as Fourier measurements, or, more generally, sensing matrices obtained by sampling a few rows 
from an orthogonal matrix. We establish that the restricted isometry property is not necessarily 
needed to accurately recover nearly sparse vectors from noisy compressive samples. This may be of 
interest because in many situations, the RIP may be hard to check, or may not hold, or does not 
hold. Thus our work is a significant departure from the majority of the literature, which establishes 
good noisy recovery properties via the RIP machinery. This literature is, of course, extremely large 
and we cannot cite all contributions but a partial list would include [15, 20, 35, 13, 16, 5, 18, 39, 
40, 3, 31, 26, 7, 2]. 

The reason why one can get strong error bounds, which are within a polylogarithmic factor 
of what is available with the aid of an 'oracle,' without the RIP is that our results do not imply 
universality. That is, we are not claiming that if A is randomly sampled and then fixed once for 
all, then the error bounds from Section 1.6 hold for all signals x. What we are saying is that if we 
are given an arbitrary x, and then collect data by applying our random scheme, then the recovery 
of this x will be accurate. 
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If one wishes to establish universal results holding for all x simultaneously, then we would need 
the RIP or a property very close to it. As a consequence, we cannot possibly be in this setup and 
guarantee universality since we are not willing to assume that the RIP holds. To be sure, suppose 
we had available an oracle informing us about the support T of x. Then we would need the pseudo- 
inverse of the submatrix with columns in T to be bounded. In other words, the minimum singular 
value of this submatrix would have to be away from zero. For a universal result, this would need to 
be true for all subsets of cardinality s; that is, the minimum singular value of all submatrices with 
s columns would have to be away from zero. This essentially is the restricted isometry property. 

To the best of our knowledge, only a few other papers have addressed non-universal stability 
(the literature grows so rapidly that an inadvertent omission is entirely possible). In an earlier 
work [9], the authors also considered weak conditions that allow stable recovery; in this case the 
authors assumed that the signal was sampled according to a random model, but in return the 
measurement matrix A could be deterministic. In the asymptotic case, stable signal recovery has 
been demonstrated for the Gaussian measurement ensemble in a regime in which the RIP does 
not necessarily hold [21, 4]. In fact, the authors of [21, 4] are able to give exact limits on the 
error rather than error bounds. Aside from these papers and the work in progress [10], it seems 
that that the literature regarding stable recovery with conditions weak enough that they do not 
imply universality is extremely sparse. Finally and to be complete, we would like to mention that 
earlier works have considered the recovery of perfectly sparse signals from subsampled orthogonal 
transforms [11], and of sparse trigonometric polynomials from random time samples [32]. 

1.8 Organization of the paper 

The paper is organized as follows. In Section 2, we introduce several fundamental estimates which 
our arguments rely upon, but which also could be useful tools for other results in the field. In 
Section 3, we prove the noiseless recovery result, namely, Theorem 1.1. In Section 4, we prove our 
main results, Theorems 1.2 and 1.3. Now all these sections assume for simplicity of exposition that 
the coherence bound holds deterministically (1.5). We extend the proof to distributions obeying 
the coherence property in the stochastic sense (1.6) in the Appendix. This Appendix also contains 
another important technical piece, namely, a difficult proof of an intermediate result (weak RIP 
property). Finally, we conclude the main text with some final comments in Section 5. 

1.9 Notation 

We provide a brief summary of the notations used throughout the paper. For an m x n matrix A 
and a subset T c {1, . . . ,n}, At denotes the m x |T| matrix with column indices in T. Also, A^j 
is the i-th column of A. Likewise, for a vector v e R n , vt is the restriction of v to indices in T. 
Thus, if v is supported on T, Av = Atvt- In particular, ak,T is the vector at restricted to T. The 
operator norm of a matrix A is denoted \A\. The identity matrix, in any dimension, is denoted /. 
Further, a always refers to the i-th. standard basis element, e.g., e± - (1,0, .. . ,0). For a scalar t, 
sgn(i) is the sign of t if 1 1 and is zero otherwise. For a vector x, sgn(x) applies the sign function 
componentwise. We shall also use fj, as a shorthand for /i(-F) whenever convenient. Throughout, 
C is a constant whose value may change from instance to instance. 
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2 Fundamental Estimates 



Our proofs rely on several estimates, and we provide an interpretation of each whenever possible. 
The first estimates E1-E4 are used to prove the noiseless recovery result; when combined with the 
weak RIP, they imply stability and robustness. Throughout this section, 5 is a parameter left to 
be fixed in later sections; it is always less than or equal to one. 



2.1 Local isometry 



Let T of cardinality s be the support of x in Theorem 1.1, or the support of the best s-sparse 
approximation of x in Theorem 1.2. We shall need that with high probability, 



\A* T A T ■ 



(2.1) 



with 5 < 1/2 in the proof of Theorem 1.1 and 5 < 1/4 in that of Theorem 1.2. Put differently, the 
singular values of At must lie away from zero. This condition essentially prevents At from being 
singular as, otherwise, there would be no hope of recovering our sparse signal x. Indeed, letting 
h be any vector supported on T and in the null space of A, we would have Ax = A{x + h) and 
thus, recovery would be impossible even if one knew the support of x. The condition (2.1) is much 
weaker than the restricted isometry property because it does not need to hold uniformly over all 
sparse subsets — only on the support set. 



Lemma 2.1 (El: local isometry) Let T be a fixed set of cardinality s. Then for 5 > 0, 

S 2 



P(\\A T A T -I\\ >S) <2sexp 



In particular, if m> —ji{F) ■ s ■ \ogn, then 



n(F)s 2(1 + 5/3) 



(2.2) 



P(\\A T A T -I\\ > 1/2) <2/n. 

Note that ||.AyAr - I\\ < 5 implies that ||(^4^Ar) _1 || < 1/(1- 5), a fact that we will use several times. 

In compressive sensing, the classical way of proving such estimates is via Rudelson's selection 
theorem [33]. Here, we use a more modern technique based on the matrix Bernstein inequality of 
Ahlswede and Winter [1], developed for this setting by Gross [25], and tightened in [37] by Tropp 
and in [29] by Oliveira. We present the version in [37]. 

Theorem 2.2 (Matrix Bernstein inequality) Let {X^} e |R dxd be a finite sequence of indepen- 
dent random self-adjoint matrices. Suppose that EX^ = and \\X)~\\ < B a.s. and put 



a 



Then for all t > 0, 



k 



>t\< 2ciexp 



\a 2 + Bt/3) 



(2.3) 
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Proof Decompose A^Aj- - I as 

m m 

A* T A T - 1= m~ l Xl( a fc,TOfc,T - -0 = m ~ Yj X k, X k : = a k,T a k,T ~ I- 

k=l k=l 

The isotropy condition implies = 0, and since ||cit||| 2 ^ ^(F) ' s > we have \\X k \\ = maxdla^rll^ - 
1, 1) < ^(F) ■ s. Lastly, < = E(a ki TO>k T ) 2 - J< E(a k: Ta k t) 2 ~ ^ ll a fc,T|| 2 afc,TOfc t- However, 

E || a KT || 2 a k>T a* k T < //(F) ■ s ■ Ea fc)T a^ T = /i(F) ■ s ■ I 

and, therefore, £k E -^t ^ w ■ A t (-^) -s- 1 so that <r 2 is bounded above by m ■ //(F) ■ s. Plugging t = 5m 
into (2.3) gives the lemma. ■ 

Instead of having A act as a near isometry on all vectors supported on T, we could ask that 
it preserves the norm of an arbitrary fixed vector (with high probability), i.e. « \v\^ for 

a fixed v supported on T. Not surprisingly, this can be proved with generally (slightly) weaker 
requirements. 

Lemma 2.3 (E2: low-distortion) Let v be a fixed vector supported on a set of cardinality at 
most s. Then for each t < 1/2, 

P(||(^M-/>||, 2 > i ||^||,J<exp(-i(ty^-l) 2 ). 

The proof is an application of the vector Bernstein inequality described in the fourth estimate E4. 
It is analogous to the proof shown there and is not repeated. 

2.2 Off-support incoherence 

Lemma 2.4 (E3: off-support incoherence) Let v be supported on T with \T\ = s. Then for 
each t > 0, 

(m t 2 \ 

-MF)'T7yu)- (24> 

This lemma says that if v = x, then max^c |(A[i}, Ax)\ cannot be too large so that the off-support 
columns do not correlate too well with Ax. The proof of E3 is an application of Bernstein's 
inequality — the matrix Bernstein inequality with d = 1 — together with the union bound. 
Proof We have 

II^MHL =m&x\(ei,A*Av)\. 
Assume without loss of generality that \\v\\^ = 1, fix i e T c and write 

(ei,A*Av) = — Y g k , g k := (ei,a k a* k v}. 
m k 

Since i e T c , Eg k = by the isotropy property. Next, the Cauchy-Schwartz inequality gives \g k \ = 
\(ei,a k ) ■ (a k ,v)\ < |(e;, a k }\ \\ak,T\\ i2 - Since |(e i? a fe )| < y/ n(F) and fla^Tf^ < \Jn{F)s, we have 
< fi(F)y/s. Lastly, for the total variance, we have 

EgUKF) E(a ktT ,v} 2 = fi(F) 
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where the equality follows from the isotropy property. Hence, a 2 < ma(F), and Bernstein's in- 
equality gives 



P(\(ei,A*Av}\ >t)< 2exp 



m t 2 



2u(F) 1 + 

Combine this with the union bound over all i e T c to give the desired result. ■ 
We also require the following related bound: 

max LA^Ara L < 8. 
ieTc ii i wu 2 

In other words, none of the column vectors of A outside of the support of x should be well approx- 
imated by any vector sharing the support of x. 



Lemma 2.5 (E4: uniform off-support incoherence) Let T be a fixed set of cardinality s. For 
any < t < ^/s, 

rat 2 1 
MF)s + 4, 

In particular, if m > 8fi(F) ■ s ■ (21ogn + 1/4), then 



P [ max ||A^Ar,i II . >t|<nexp -- 
\ i(T c II i WHz ) , 



P(max||^ w || f2 >l)<l/n. 



The estimate follows from the vector Bernstein inequality, proved by Gross [25, Theorem 11]. 
We use a slightly weaker version, which we find slightly more convenient. 



Theorem 2.6 (Vector Bernstein inequality) Let {v k } e R be a finite sequence of independent 

9 ii ii 2 

random vectors. Suppose that Ev k = and H^fcH^ < B a.s. and put a > Y,k^-\\ v k\\£ 2 - Then for all 
0<t<a 2 /B, 



PI 



k 



> t 

(> 



Note that the bound does not depend on the dimension d. 
Proof Fix i e T c and write 

m i m 

A T A {i} = — E a lA a k, di) := — Yj v k- 

m j=1 m k=l 

As before, Ev k = Ea* kT (a k ,ei) = since i e T c . Also, \\v k \\ i2 = ||afe,r|| f2 [(afcj e^)] ^ v(F)^fs. Lastly, 
we calculate the sum of expected squared norms, 

m 

^ 2 2 2 2 2 

2] E lujfcl^ = mE ||vi || ^ 2 < mE[||ai )T ||^ (e»,ai) ] < mn(F)s ■ E(ej,ai) = m(i(F)s. 
k=i 

As before, the last equality follows from the isotropy property. Bernstein's inequality together with 
the union bound give the lemma. ■ 
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2.3 Weak RIP 



In the nonsparse and noisy setting, we shall make use of a variation on the restricted isometry 
property to control the size of the reconstruction error. This variation is as follows: 

Theorem 2.7 (E5: weak RIP) Let T be a fixed set of cardinality s and fix 5 > 0. Then for all v 
supported on T u R, where R is any set of cardinality \R\ < r, we have 

(1 ~S) \v\\ <\\Av\\< (i + a)H|? a (2-6) 

with probability at least 1 - 5e _/3 provided that 

m > C$ ■ (3 ■ n(F) ■ max(s log n,r log 5 n). 

Here C$ is a fixed numerical constant which only depends upon S. 

This theorem is proved in the Appendix using Talagrand's majorizing measures theorem, and 
combines the framework and results of Rudelson and Vershynin in [35] and [33]. In the proof of 
Theorem 1.2, we take 5 = 1/4. 

The condition says that the column space of At should not be too close to that spanned by 
another small disjoint set R of columns. To see why a condition of this nature is necessary for 
any recovery algorithm, suppose that x has fixed support T and that there is a single column A^y 
which is a linear combination of columns in T, i.e., A Tu ^y is singular. Let h t be supported on 
T u {i} and in the null space of A. Then Ax = A{x + th) for any scalar t. Clearly, there are some 
values of t such that x + th is at least as sparse as x, and thus one should not expect to be able 
to recover x by any method. In general, if there were a vector v as above obeying ||^4u||^ 2 « 
then one would have A?vt ~ -Arvr. Thus, if the signal x were the restriction of v to T, it would 
be very difficult to distinguish it from that of —v to R under the presence of noise. 

The weak RIP is a combination of the RIP and the local conditioning estimate El. When r = 0, 
this is El whereas this is the restricted isometry property when s = 0. The point is that we do 
not need the RIP to hold for sparsity levels on the order of m/[fi{F) logra]. Instead we need the 
following property: consider an arbitrary submatrix formed by concatenating columns in T with r 
other columns from A selected in any way you like; then we would like this submatrix to be well 
conditioned. Because T is fixed, one can prove good conditioning when s is significantly larger than 
the maximum sparsity level considered in the standard RIP. 

2.4 Implications 

The careful reader may ask why we bothered to state estimates E1-E4 since they are all implied by 
the weak RIP! Our motivation is three-fold: (1) some of these estimates, e.g. E2 hold with better 
constants and weaker requirements than those implied by the weak RIP machinery; (2) the weak 
RIP requires an in-depth proof whereas the other estimates are simple applications of well-known 
theorems, and we believe that these theorems and the estimates should be independently useful 
tools to other researchers in the field; (3) the noiseless theorem does not require the weak RIP. 

3 Noiseless and Sparse Recovery 

This section proves the noiseless recovery theorem, namely, Theorem 1.1. Our proof essentially 
adapts the arguments of David Gross [25] from the low-rank matrix recovery problem. 
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3.1 Dual certificates 

The standard method for establishing exact recovery is to exhibit a dual certificate; that is to say, 
a vector v obeying the two properties below. 

Lemma 3.1 (Exact duality) Set T = supp(x) with x feasible for (1.9), and assume At has full 
column rank. Suppose there exists v e R n in the row space of A obeying 

vt = sgn(xx) and \\'^T c \\£ go < 1- (3-1) 
Then x is the unique t\ minimizer to (1.9). 

The proof is now standard, see [15]. Roughly, the existence of a dual vector implies that there is a 
subgradient of the i\ norm at x that is perpendicular to the feasible set. This geometric property 
shows that x is solution. Following Gross, we slightly modify this definition as to make use of an 
'inexact dual vector.' 

Lemma 3.2 (Inexact duality) Set T = supp(x) where x is feasible, and assume that 

\{A* T A T y 1 \<2 and max \A* T A {i} ||^ < 1. (3.2) 

Suppose there exists v e R n in the row space of A obeying 

\\vt - sgn{xr)\i 2 < 1/4 and \ v t4i„ ^ 1/4- (3.3) 
Then x is the unique i\ minimizer to (1.9). 

Proof Let x = x + h be a solution to (1.9) and note that Ah = since both x and x are feasible. 
To prove the claim, it suffices to show that h = 0. We begin by observing that 

11% = \\ x t + hr\ tl + \\hr4e! * Nrll^ + (sgn(x T ),h T ) + 11%% ■ 
Letting v = A*w be our (inexact) dual vector, we have 

(sgn(x T ),/iT) = (sgn(a?r) — vt, hr) + (v T ,h T ) = (sgn(xT) -vt,Ht) - {vT<^,h T c), 

where we used (vT,hr) = (v,h) - (t>T c ,^T c } = -(t>T c ,^r c ) since (v,h) = (w,Ah) - 0. The Cauchy- 
Schwartz inequality together with the properties of v yield 

|(sgn(a; T ),/j T )|<-(||/i T ||^ + ||/ ir c|| 4 ) 

and, therefore, 

1 3 

11% * ll^-^IIMU + ^IIMU- 

We now bound ||/it||£ 2 - First, it follows from 

h T = (A^At) 1 A T Axh = -{A^A T y l A* T A T ch T c 
that || hx \\i 2 < 2 || A^At<: hx c \ t ■ Second, 

\\ A T A T°h T 4 h < 2 \\ A T A {i}\\ h \hi\< max|[A^4 { ; } || \\hr4h ^ W h T4e 1 ■ 
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In conclusion, || /it || 2 ^ 2||/itc||i and thus, 

This implies /it c = 0, which in turn implies Jit = since we must have A?hx = Ah = (and At has 
full rank). ■ 



Lemma 3.3 (Existence of a dual certificate) Under the hypotheses of Theorem 1.1, one can 
find v e IR" obeying the conditions of Lemma 3.2 with probability at least 1 - - 1/n. 

This lemma, which is proved next, implies Theorem 1.1. The reason is that we just need to 
verify conditions (3.2). However, by Lemmas 2.1 and 2.5, they jointly hold with probability at least 
1 - 3/n provided that m> fi- s- (191ogn + 2) (recall that \x is a shorthand for fi(F)). 



3.2 Proof of Lemma 3.3 

The proof uses the clever golfing scheme introduced in [25]. Partition A into row blocks so that 
from now on, Ai are the first mi rows of the matrix A, A<i the next rn-2 rows, and so on. The £ 
matrices {^4j}| =1 are independently distributed, and we have mi + rri2 + . . . + mi = m. As before, 
A^t is the restriction of Ai to the columns in T. 

The golfing scheme then starts with vq = 0, inductively defines 

m 

Vi = — A*A i T (sgn(x T ) - Vi- 1T ) + 

TTH 

for i = 1,...,£, and sets v = ve- Clearly v is in the row space of A. To simplify notation, let 
qi = sgn(xT) - Vt,Tj and observe the two identities 

(TTb \ ^ / TTL \ 
/ A* tT A i:T qi -i = J] / A) T Aj. T sgn(2; T ) (3.4) 
m i I j=l \ m j I 

and 

/ 

v = Y, — A*A itT q^i, (3.5) 

which shall be used frequently. From (3.4) and the fact that I - ^-A* T Ai^T should be a contraction 
(local isometry El), we see that the norm of % decreases geometrically fast — the terminology 
comes from this fact since each iteration brings us closer to the target just as each golf shot would 
bring us closer to the hole — so that vt should be close to sgn(xj'). Hopefully, the process keeps 
the size of vt<= under control as well. 

To control the size of vt c and that of sgn(xT) ~vt, we claim that the following inequalities hold 
for each i with high probability: first, 

hi \\t 2 ^Cilk-i \\t 2 (3.6) 

and, second, 

m A* A 
— A i T cA i,T Qi-l 
mi ' 



< k hi 



(3.7) 
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(the values of the parameters ij and q will be specified later). Let pi(i) (resp. p 2 (i)) be the 
probability that the bound (3.6) (resp. (3.7)) does not hold. Lemma 2.3 gives 



Thus, if 



pi(i) < exp^-j(a\/mi/(sfi) - l) 2 j 



2 + 8(/? + loga) 
mi > = S A*' 



then pi(i) < —e ? . Next, Lemma 2.4 gives 



P2(i) ^ 2nexp - 



Thus, if 



6/i + 2/jLy/sti ) 

(j3 + log(2a) + logn)s/i, 



(3.8) 
(3.9) 

(3.10) 
(3.11) 



then p%{i) < ■ 

It is now time to set the number of blocks £, the block sizes rrii and the values of the parameters 
Cj and ti. These are as follows: 

• €=[(log 2S )/2] + 2; 

• ci = C2 = l/[2\/logn] and q = 1/2 for 3 < i < £; 

• ti=t 2 = l/[8v/s] and U = logn/[8y/s] for 3 < i < £; 

• mi, m2 > 35(1 + log 4 + f3)s/j,a[ 2 and mi > 35(1 + log 6 + (3)sficJ 2 for 3 < i < £. 

It is not hard to see that the total number of samples m = Y,i m i obeys the assumptions of the 
lemma. To see why v is a valid certificate, suppose first that for each i, (3.6) and (3.7) hold. Then 
(3.4) gives 

||sgn(x T ) -v T \\ h = \\qe\\ t2 < ||sgn(x T )|| <2 a < < - 



as desired. Further, (3.5) yields 



111: 



AiTcAi t Tqi-i 



l l i-l 

i=i j=i 



i=l 



Now with our choice of parameters, the right-hand side is bounded above by 



1 / 1 

- 1 + ; 

8 \ 2v^oi7T 



logn 
g n 4 log n 



+ ••• < 



1 



which is the desired conclusion. 

Now we must show that the bounds (3.6), (3.7) hold with probability at least 1 - - 1/n. 
It follows from (3.9) and (3.11) that Pi(i),P2(i) ^ \ e ~^ f° r * ~ 1, 2 and Pi(i),P2(i) ^ \ e ~^ - 
for i > 3. Thus, pi(l) + pi(2) +^(1) + P2(2) < e _/3 and + p 2 (i) < 1/3 for i > 3. Now the 
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union bound would never show that (3.6) and (3.7) hold with probability at least 1 - 1/n for all 
i > 3 because of the weak bound on p\{i) + p2(i)- However, using a clever idea in [25], it is not 
necessary for each subset of rows to 'succeed' and give the desired bounds. Instead, one can sample 
a 'few' extra batches of rows, and throw out those that fail our requirements. We only need £ — 2 
working batches, after the first 2. In particular, pick £' + 2 > £ batches of rows, so that we require 
m > 2 • [140(1 + log 4 + (3) ■ \i ■ s ■ logn] + 1 ■ [140(1 + log 6 + /3)sfj,] (note that we have made no attempt 
to optimize constants). Now as in [25], let N be the the number of batches — after the first 2 - 
obeying (3.6) and (3.7); this ./V is larger (probabilistically) than a binomial (£' , 2/3) random variable. 
Then a standard concentration bound [28, Theorem 2.3a] 

P(N <£-2) <expl-2-^ -I 

tells us that if we were to pick £' = 3[logn] + 1, we would have 

P(iV<^-2) < 1/n. 

In summary, from pi(l) + P2{1) + pi(2) + p2(2) < and the calculation above, the dual 
certificate v obeys the required properties with probability at least 1 - 1/n - , provided that 
m > C(l + j3) • p, ■ s ■ log n. 



4 General Signal Recovery from Noisy Data 

We prove the general recovery theorems from Section 1.6 under the assumption of Gaussian white 
noise but would like to emphasize that the same result would hold for other noise distributions. 
Specifically, suppose we have the noisy model 

y-Ax + z, where < A n (4-1) 

holds with high probability. Then the conclusions of Theorem 1.3 remain valid. In details, the 
Dantzig selector with constraint ((^""(y - Ax)]]^ < 4A„ obeys 



||x-x||^ 2 < Ci(l + a 2 ) 



(4.2) 



with high probability. Hence, (1.15) is a special case corresponding to A n = 2.5a m v / log n = 

2.5a^J^^. Likewise, the bound on the i\ loss (1-16) with \ n in place of <J \f^^ holds as well. 
A similar generality applies to the LASSO as well, although in this case we need a second noise 
correlation bound, namely, 

\\A^ c (I~P)z\\ u <\ n . 
Now when z ~ A/"(0, /) and A is a fixed matrix, we have 

||A*z|| /oo <2||i4||i,2v^Ogn (4-3) 
with probability at least 1 - l/2n; here, 2 is the maximum column norm of A. Indeed, the ith 

II I|2 

component of A* z is distributed as Af(0, \\A{i}\\ i2 ) an d, therefore, the union bound gives 
P(\\A*z\\ eoo > 2||4|li | 2v / iogn) <rcP(|Af(0, 1)| >2 v / lo^). 
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The conclusion follows for n > 2 from the well-known tail bound P(|A/"(0, 1)| > t) < 2<f>(t)/t, where 
<j) is the density of the standard normal distribution. The same steps demonstrate that 

\\A*(I - P)z\\ £x <2\\(I - P)A\\ 1)2 y/^<2\\A\\ 1>2 y/^ (4.4) 

with probability at least 1 - l/2n. 

4.1 Proof of Theorem 1.2 

We begin with a few simplifying assumptions. First, we assume in the proof that a m = 1 since the 
general result follows from a simple rescaling. Second, because we are interested in situations where 
m is much smaller than n, we assume for simplicity of presentation that m<n although our results 
extend with only a change to the numerical constants involved if m < n°^ l \ In truth, they extend 
without any assumption on the relation between m and n, but the general presentation becomes a 
bit more complicated. 

Fix s obeying s < s, and let T = supp(x s ). We prove the error bounds of Theorem 1.2 with s 
fixed, and the final result follows by considering that s which minimizes either the (.2 (1-11) or l\ 
(1.12) error bound. This is proper since the minimizing s has a deterministic value. With T as 
above, we assume in the rest of the proof that 

(i) all of the requirements for noiseless recovery in Lemma 3.2 are met, 

(ii) and that the inexact dual vector v of Section 3 is successfully constructed. 
All of this occurs with probability at least 1 - 4/n - eT^ . Further, we assume that 

(iii) the weak RIP holds with 5 = 1/4, r = -77- n . ; — 5— v s and T is as above. 

This occurs with probability at least l-5e~^, and implies the RIP at sparsity level r and restricted 
isometry constant 5 = 1/4. Lastly, we assume 

(iv) the noise correlation bound 

\\A*z\\ ioo < 2.5A/logn. (4.5) 

Assuming the weak RIP above, which implies ||-A||i,2 < 5/4, the conditional probability that this 
occurs is at least l-l/2n because of (4.3). Because the weak RIP implies the local isometry condition 
El with 5 = 1/4, all of these conditions together hold with probability at least l-4/n-6e~^. All of 
the steps in the proof are now deterministic consequences of (i)-(iv); from now on, we will assume 
they hold. 

With h - x- x, our goal is to bound both the £2 and l\ norms of h. We will do this with a pair 
of lemmas. The first is frequently used (recall that A is set to lO^logn). 

Lemma 4.1 (Tube constraint) The error h obeys 

\\A*Ah\\ ioa < ^. 
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Proof As shown in [9, Lemma 3.1], writing that the zero vector is a subgradient of the LASSO 
functional ^ II 2/ _ -^lll 2 + A||5||4 at cc = :£ gives 

\\A*{y-Ax)\\ ioo <\. 

Then it follows from the triangle inequality that 

\\A*Ah\\ laQ < \\A*{y-Ax)\\ loo + \\A*z\\^ < A + \\A*z\\ ioo , 

where z is our noise term. The claim is a consequence of (4.5). ■ 



Lemma 4.2 The error h obeys 



for some numerical constant Cq. 



I^HI^ ^ C (sA + ||xtc||^) 



(4.6) 



Before proving this lemma, we show that it gives Theorem 1.2. Some of the steps are taken from 
the proof of Theorem 1.1 in [16]. 

Proof [Theorem 1.2] Set r as in (iii) above. We begin by partitioning T c and let T\ be the indices 
of the r largest entries of h^, 2~2 be those of the next r largest, and so on. We first bound ||/ituTi ||^ 2 
and set T\ = T u T\ for short. The weak RIP assumption (iii) gives 



7 II^TiLa ~ l^ T i^ T il^2 ~ ^Ti^Ti' Ah) (A Ti h Ti ,A T ch T c). 



(4.7) 



From Lemma 4.1, we have 

(A Ti h Tl ,Ah) = (h fi ,A* fi Ah) < ||/tfj 4 



A* Ti Ah 



< -A \\hn 



[ 1 



Since T\ has cardinality at most 2s, the Cauchy-Schwartz inequality gives 

(A^h^Ah) < -\V2^\\h Tl \\ £2 . 

Next, we bound (A^h^, Afchfc) < [Axhx , Afchfc) + (A^h^ , Afchfc) 

{A T h T ,A f ch f c} < Y, \{A T h T ,A T .h T .)\ . 

i>2 



. We have 



(4.8) 



(4.9) 



As shown in [14, Lemma 1.2], the parallelogram identity together with the weak RIP imply that 



1 



\{A T hr,A Ti hr } )\ < - HMI& \hr. 



i\\£ 3 



and, therefore, 



1 



(A T h T , A f? h f c) < - ||/i T || 4 Y, \\hTjL 
4 i>2 



To bound the summation, we use the now standard result [16, (3.10)] 

EINL ^ r_1/2 II Ml*, 

i>2 



(4.10) 



(4.11) 
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which gives 



(A T h T ,A t ch f c)\ < V 1 / 2 \\h T \\ l2 \\h T c\\ h . 
The same analysis yields (A^h^, Afchfc) < \r~ l l 2 \\hxx \\i 2 Whr^W^ and thus, 



1 



-1/2 



(A Ti h Ti , A T c h T c } 
Plugging these estimates into (4.7) gives 

ii^ 1 i, 2 4(^ A+r_i/2 ii /irc ^) 

The conclusion is now one step away. Obviously, 



(4.12) 



Elk 



i>2 



)\\e 2 * \\ h Tih 2 
1/5 



+ r 



■ 1/2 \\hr° 



J-VTsX + Sr' 1 / 2 \\h T 4 £i ), 



2\2 

where the second line follows from (4.12). Lemma 4.2 completes the proof for the £2 error. For the 
i\ error, note that by the Cauchy- Schwartz inequality 

Combine this with (4.12) and Lemma 4.2. ■ 

4.2 Proof of Lemma 4.2 

Since x is the minimizer to (1.10), 

2 \\ Ax -y\\j 2 + x\\x\\ ei * 2 W Ax ~y\\h + x W x hi > 

which can be massaged into the more convenient form 

1 2 

- \\Ah\\ h + X \\x\\ £i < (Ah, z) + X \\x\\ £i . 



Lemma 4.3 



h > \\ x \\e 1 + (^T,sgn(x T )) + \\hrc\\ h - 2 \\x T 4i 1 



Proof We have \\x\\ ei = (x, sgn(x)) > (xy + hx, sgn(xr)) + \\xt c + hr c \^ and the claim follows from 
the triangle inequality. ■ 



It follows from this that 

1 2 

2 \\Ah\\t 2 + M\h T c\\ h < (Ah,z) - X(h T ,sga(x T )} + 2X\\x T 4e 1 
and the proof is now a consequence of the two short lemmas below. 



(4.13) 
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Lemma 4.4 



(Ah,z)<^s\ 2 + \\\h T 4 h . (4.14) 



Proof The proof is similar to an argument in [9]. Let P = At{A t At) l A T be the orthogonal 
projection onto the range of At- Then 

{Ah, z) = (PAh, z) + ((I- P)A T ch T c,z) 

= (A T Ah, {A* T A T y l A T z) + (h T c,A TC (I- P)z) 

< \\A T Ah\\^ \\{A T A T y l A T z\\ h + \\h T o\\ ti \\A Tc (I-P)z\\ loo 

<-\\\{A T A T )' l A T z\\ + 2.5^1 ogn\h T c\„. (4.15) 
The last line follows from Lemma 4.1 and (4.4). We now bound the first term, and write 

<\^\\A T z\\ i2 

<\s\\A T z\\ u < l -s\. (4.16) 

The first inequality follows from Cauchy-Schwartz, the second from HA^^^H < 4/3, and the fourth 
from 11^4*^1^ < A/4. Inequality (4.15) establishes the claim. ■ 

Lemma 4.5 

\(h T , sgn(x T ))| < CsX+ ^ \\h T o\\ k + ^ \\Ah\] 2 . (4.17) 

Proof Let v be the inexact dual vector, and decompose (Ht, sgn(xT)} as 

| {h T , sgn(x T )} | < \(h T ,sgn(x T ) -v T )\ + \(h T ,v T )\ 

< \(h T ,sgn(x T )-v T )\ + \(h,v)\ + \(h T c,v T c)\. (4.18) 

First, 

\(h T ,sgn(x T )-v T )\ < \\h T \\ i2 \\sgn(x T ) - v T \\ h < \ \h T \ h ■ 

Now 

IIMk * \{A T A T y x \ \\A T A T h T \\ e2 < \ \\A T A T h T \\ h 

< -\\A T Ah\\ t2 + -\\A T A T ch T 4t 2 

< ^\\A T Ah\\ eoo + - \\h T 4 h mzx\\A T A {j} \\ h 

<^A + ^||/t T c|| 4 , (4.19) 

where the last line follows from Lemma 4.1 and (3.2). Second, it follows from the definition of v 
that 

\(hre,VT°)\ < l^Hk II^HI^, ^ \ U^Hl* • 
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Hence, we established 



\(h T , S gn(x T ))\ < ^ S \ + L \\ hTC \\ h + |( M )| . (4.20) 



Third, we bound \(h, v)\ by Lemma 4.6 below. With the notation of this lemma, 

|(7i,u) | = \(h,A*w)\ = \(Ah,w}\ < \\Ah\\ t2 lw\\ ta < C y^\\Ah\\ t2 
for some Co > 0. Since 

,,,,,, r , \ Ah \\ C s\ 

\\Ah\\ p Js < - + , 

11 2C A 2 ' 

it follows that 



l(MI<^A + ^||A/4 2 2 . (4.21) 
Plugging this into (4.20) finishes the proof. 



Lemma 4.6 The inexact dual certificate from Section 3 is of the form v = A*w where \ w\i < Cq\J~s 
for some positive numerical constant Cq. 

Proof For notational simplicity, assume without loss of generality that the first £ batches of rows 
were those used in constructing the dual vector v (none were thrown out) so that 

i 

v = Y, — A*A iT qi-i. 
i=i m i 

Hence, v = A*w with w* = (w^, . . . ,w%, 0, . . . ,0) and Wi ■= ^-Ai^qi-i so that \\w\\^ 2 = Yd=i \\ w i ||j? 2 - 
We have 



^7 \\A,Tqi-ih n = (^rAl T A i;T qi-i,qi-i) 



£2 > mi ' 

= ( {^-A* T A i)T - , ) + || || \ 

^ \\li\\l 2 lki-l||^ + lki-lb 2 
2 



^ 2 ki-ill^ 



It follows that 



i-1 

<2s]~[ c i- ( 4 - 22 ) 

3=1 



1, 2 „ „ 4* m T-T 2 



i=l l'h j=l 



Assume that m < C(l + /3)/islogn so that m is just large enough to satisfy the requirements of 
Theorem 1.2 (up to a constant). Then recall that mi > C(l + f3)fisc i 2 => ^- < Ccflogn. (If m is 
much larger, rescale each nii proportionally to achieve the same ratio.) This gives 



Mil < c s iogn y, n c ' * Cs e n c i 



i=lj=l i=lj=2 

since c\ = (2^/Iogn) -1 . For i> 1, Yl)=2^~ an d the conclusion follows. 
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4.3 Proof of Theorem 1.3 

Proof Fix s and T as in Section 4.1 and assume that (i)-(iv) hold. The proof parallels that for 
the LASSO; this is why we only sketch the important points and reuse the earlier techniques with 
minimal extra explanation. We shall repeatedly use the inequality 

ab<ca 2 /2 + b 2 /(2c), (4.23) 
which holds for positive scalars a,b,c. Our first intermediate result is analogous to Lemma 4.2. 
Lemma 4.7 The error h-x-x obeys 

\h T o\ h < C(sA+ \\x T 4i 1 + Vs\\Ah\\ h ). 
Proof Since x is feasible, H^H^ < \\x\\g and it follows from Lemma 4.3 that 

H^HI^ <-(hT,sgn(x T ))+2\\x T 4i 1 - ( 4 - 24 ) 

We bound \(hT, sgn(xr))| in exactly the same way as before, but omitting the last step, and obtain 

7 

|(/i T ,sgn(x T )}| < Cs\ + — + Csfs\Ah\ l2 . 

This concludes the proof. ■ 

The remainder of this section proves Theorem 1.3. Observe that |.A*A/i|& xl < |A (Lemma 4.1) 
since the proof is identical (we do not even need to consider subgradients). Partitioning the indices 
as before, one can repeat the earlier argument leading to (4.12). Then combining (4.12) with 
Lemma 4.7 gives 

\h Ti \ l2 < C^~s\ + Cr- ll2 (s\ + \\x T A\ ei + V~s\\Ah\\i 2 )- (4-25) 

The term proportional to \fsfr\Ah\g in the right-hand side was not present before, and we must 
develop an upper bound for it. Write 

\\Ah\l = (A*Ah,h) < \\A*Ah\\ igo \\h\\ £l < h(\\h T \\ fl + \\h T c\\ h ) 

and note that (4.24) gives 

\\h T c\\ ei < \\h T \\ ei +2\\x T c\\ ei . 

These last two inequalities yield ||A/i|^ a < fAd^rl^ + l^T 13 !^)) and since -J A \\xt<= < ^X\/s + 
^7 ll^iHI^ because of (4.23), we have 



\ Ah \\i 2 < yJ^iy/WhrWei+yJlMleJ * V^V^IM* + §V^ + jjs IMli,)- 



In short, 



<c[Vs\ + r 1/2 (sA+ \\x T 4e 1 + y/sX\\hr\\ ei )). 



The extra term on the right-hand side has been transmuted into C^J^X fhrf^, which may be 
bounded via (4.23) as 

cJ'-X\\h T \\ ei < C 2 - V~s\ + \\h T \\ ei < C 2 - yT sX + I || M . 
V r 1 r 2^s r 2 2 
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Since ||/ir||^ 2 < |^fil^ 2 > we nave 

lfel fe ^( 1+ yi + £)viA +C ^. 

The remaining steps are the same as those in the proof for the LASSO. ■ 

5 Discussion 

This paper developed a very simple and general theory of compressive sensing, in which sensing 
vectors are drawn independently at random from a probability distribution. In addition to estab- 
lishing a general framework, we showed that nearly sparse signals could be accurately recovered 
from a small number of noisy compressive samples by means of tractable convex optimization. For 
example, s-sparse signals can be recovered accurately from about s log n DFT coefficients corrupted 
by noise. Our analysis shows that stable recovery is possible from a minimal number of samples, and 
improves on previously known results. This improvement comes from novel stability arguments, 
which do not require the restricted isometry property to hold. 

We have seen that the isotropy condition is not really necessary, and it would be interesting 
to know the extent in which it can be relaxed. In particular, for which values of a and j3 obeying 
al< Eaa* < (31 would our results continue to hold? Also, we have assumed that the sensing vectors 
are sampled independently at random, and although the main idea in compressive sensing is to use 
randomness as a sensing mechanism, it would be interesting to know how the results would change 
if one were to introduce some correlations. 



A Proof of Theorem 2.7 (the weak RIP) 

Our proof uses some the results and techniques of [33] and [35]. Recall that A is a matrix with 
rows drawn independently from a probability distribution F obeying the isotropy and incoherence 
conditions, and that we wish to show that for any fixed < 5 < 1, 

(i-*) \Ml<\\Ml<^ + 5) 

These inequalities should hold with high probability, uniformly over all vectors v obeying supp(u) c 
T u R where T is fixed, R may vary, and 

\T\ <c— , \R\<c- 



[i log m fi log n log m 

To express this in another way, set 

X := sup 1 1| || | 2 - \\v\\j 2 \ , 

veV 

where 

V = {v: \\v\\ h = 1, supp(u) c T u R, \R\ < r, T n R = 0}. (A.l) 

In words, v is a unit-normed vector supported on T u R where T is fixed of cardinality s < 
cm/(filogm), and R is any set disjoint from T of cardinality at most r < cm/(fi log n log 4 m). 
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We wish to show that X < 5 with high probability. We will first bound this random variable in 
expectation and then show that it is unlikely to be much larger than its expectation. The bound 
in expectation is contained in the following lemma. 

Lemma A.l Fix e > 0. Suppose m > C /j,[s log m v r log n log 4 m] , where C is a constant only 
depending on e. Then 

EX <e. 

To begin the proof, note that for any v with supp(u) cTuiJ, we have 

\\Av\\l = \\A T v T \\l + \\A R v R \\l + 2{v T ,A* T A R v R }. 

The first two terms are easily dealt with using prior results. To be sure, under the conditions of 
Lemma A.l, a slight modification of the proof of Theorem 3.4 in [35] gives 2 

E sup HlAflUfllJ - flufllj | < 7lM|< a . (A.2) 

v R :\R\<r 4 
Next, it follows from [34], or the matrix Bernstein inequality in Estimate 1, that 

Esup|||A T <u T ||| - \\v T \\ 2 h \ < - \\v T \\j 2 . (A.3) 

vt 4 

Thus, to prove Lemma A.l, it suffices to prove that 

E max || A T || < e/4. 

R. 

This is the content of the following theorem. 
Theorem A.2 Under the assumptions of Lemma A.l, we have 

ir ll a* A l s ^°& m , , / r/xlogrelog^ 

Emax || A r At \\ < C \ \ + \ 

R \ V rn V m 

Put differently, the theorem develops a bound on 

^ m 

E max — Y(ai,x){ai,y) (A.5) 

(x,y)<EBxD m i=1 

in which 

B ■- {x ■ \\x\\ i2 < l,supp(x) c T}, 

D -{y- \\y\\e 2 ^ 1, supp(y) n T = 0, |supp(y)| < r}. 

By symmetrization followed by a comparison principle - both of which follow by Jensen's inequality 
(see [27, Lemma 6.3] followed by [27, inequality (4.8)]), (A.5) is less or equal to a numerical constant 
times 

rn 

E max — Ygi(ai,x)(ai,y), 

(x,y)zBxD m 

where the g^s are independent A/"(0, 1) random variables. The main estimate is a bound on the 
conditional expectation of the right-hand side; that is, holding the vectors at fixed. 

2 Rudelson and Vershynin consider a slightly different model but the proof in [35] extends to our model with hardly 
any adjustments. 



(A.4) 
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Lemma A. 3 (Main lemma) Fix vectors and let 

Ri := max — y^(a,i,x) 2 , R2 ■= max — 

xeB m i=1 y^D m i=1 

Suppose m> C fi[s log mv r log n log 4 m] . Then 

1 ™ / /(l + i2 2 )|T|/ilogm /(l + i2i)s//lognlog 3 m\ 
E max — £g i (a i ,x)(a i ,y)<C' a/^ +\/- ^-^^ 5 . 

Proof [Theorem A. 2] Under the assumptions of the theorem, it follows from the results in [35] 
and Jensen's inequality that E %/ 1 + R2 S= a/1 + E R2 < C. Likewise, the results in [34] and the same 
Jensen's inequality give E \/l + R~\ < C. (These inequalities were also noted, in a different form, in 
(A. 3) and (A. 2)). Hence, Lemma A. 3 implies 

r w a* a \\ s n\ fsjj^ogm / r/i log n log 3 m \ 

Lmax AdAt < C a / h \ • 

R \ V rn V m J 



A.l Proof of Lemma A. 3 

We need to develop a bound about the expected maximum of a Gaussian process, namely, 

E max F(x,y), 

(x,y)zBxD 

where 

m 

4=1 

We shall do this by means of the majorizing measure theorem below, which may be found in [33] 
and is attributed to Talagrand (combine Theorem 4.1 with Propositions 2.3 and 4.4 in [36]). From 
now on, (M, d) is a metric space and B(t, e) is the ball of center t and radius e under the metric d. 

Theorem A. 4 (Majorizing measure theorem) Let {Xt)teM be a collection of zero-mean ran- 
dom variables obeying the subgaussian tail estimate 

F{\X t - AV| > u) < exp l-c-^-^j , (A.6) 

for all u > 0. Fix p > 1 and let ko be an integer so that the diameter of M is less than p~ k ° . 
Suppose there exist a > and a sequence of functions Wk}"^^' ■ M -> IR + , with the following 
two properties: 1) the sequence is uniformly bounded by a constant depending only on p; 2) for each 
k and for any t e M and any points ti,—,tfi 6 B(t,p~ k ) with mutual distances at least p~ k ~ l , we 
have 

max tp k+2 (tj) > ip k (s) +o-p~ k \J\ogN. (A.7) 

Then 

EsupA t < C(p) -a' 1 . (A.8) 
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To apply this theorem, we begin by bounding the variance between increments in order to 
ascertain the metric we need to use. We compute 



ill i i \ 2 

Yav(F(x,y) - F(x',y')) = ^(( a i> v){<k, y) ~ {ai,x')(ai,y'}j 

i=l 
m 

= E(( a i' x ~ x '){ a iiy) -(ai,x')(ai,y' -y}) 



i=l 

\ui, x - x i\Ui, y/ - \Ui,x /\Ui, y - y/ 2 

i=l 

m m 

r\2 



< 2Y / {ai,x-x') 2 (a i ,y) 2 + 2Y / {ai,x} 2 (a i ,y-y') 

i=l i=l 
ii 2 ii 2 

< 2 \\x - x'\\ B + 2mRi \\y - y'\ D , 



where we define the norms , as follows: 



x\\ B ■- max 



nax \ ? 



m 

Y,{ai,x) 2 {ai,y) 2 , \\y\\ D := max |(ai,y>| . 

l<i<m 



(We note that they may be pseudo norms, but this makes no difference to the proof. All of the 
utilized lemmas and theorems generalize to pseudo norms.) Thus, since \/c + d < \fc + \fd for any 
scalars c, ci, we can use the metric 



d((x,y),(x',y')) := \/2 \\x - x'\\ B + y/2mR 1 \\y-y'\\ D . 

Before continuing, we record two useful lemmas for bounding N. Here and below, N(M, d, e) 
is the covering number of M in the metric d. 

Lemma A. 5 (Packing number bound) Let ti,t2,—,t^ e M be points with mutual distances at 
least 2e under the metric d. Then 

N < N(M,d,e). 

This is a standard result proved by creating an injective mapping from the points {tj} to those in 
the cover set (map each tj to the nearest point in the cover). 

The next lemma is a standard tool used to obtain bounds on covering numbers, see [30] and [6] 
for a more general statement. 

Lemma A. 6 (Dual Sudakov minorization) Let Bi 2 be the unit £2 ball in R d , and let ||-|| be a 

norm. Let z 6 R d be a Gaussian vector with independent Af(0, 1) entries. Then there is a numerical 
constant C > such that 

C 



^\ogN{B l2 ,\\-\\,e)<-\fi 



11 2 
z\\ . 

e 



We now invoke the majorizing measure theorem to prove Lemma A. 3. We start by bounding 
the diameter of BxD under the metric d. For any x e B, \(a,i,x)\ < Ha^T^ and, likewise, for 

any y e D, |(aj, y)| < sjrjx. This gives 1) |cc - x'\ B < 2\J rsfi 2 m for any x, x' e B, 2) s/mRi < ^/sjlrn, 
and 3) \\y - y'Wp < 2^/rJl for any y,y' e D. Combined, these bounds yield 

d((x,y), (x' ,y')) < bfi\/rsm. 
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Under the stated assumptions, the right-hand side is at most to 3 / 2 and we thus set ko to be the 
largest integer such that 

p- ko >m 3/2 . 

We now define (pk on coarse and fine scales. In what follows, we may take p = 6 so that C{p) 
(A. 8) is an absolute constant. 

Coarse scales: for k = ko, ko + 1, . . . ,0, 

/ \ r ii ii 2 II ii — fc-i ^ — ^0 

ipk{x) := mm{ it L : it - x \\ B < p } + . 

z logn 

Fine scales: for k > 1, is a constant function given by 

ip k (x) := 3pcr T 1 JlogN(B x D,d,e)de + 3. 
Jp- fc 

Lastly, set 

o" -1 := C\fm (1 + R2)sp\ogm + \J (1 + R\)sp logn log 3 mj . 

Our definition of is closely related to-and inspired by-the functions defined in [33] . We need to 
show that these functions are uniformly bounded and obey (A. 7) for all k. We begin by verifying 
these properties for fine scale elements as this is the less subtle calculation. 

A. 2 Fine scale: k > 1 

To show that (A. 7) holds, observe that, 



+2 - <Pk = 3ap f ^ \/log N(B x D,d,e)de 

Jp-(k+2) 

> 3ap J^ k+2) ^logN(BxD,d,e)de 

> 3a p( l -p-^ - p-V* + Q)^]0gN(BxD,d,±p-V«-i)) 



> ap k \/logN. 



The last line follows from p > 6 and the packing number bound (Lemma A. 5). Note that this same 
calculation holds when k = 0,-1 because for k < 0, <pk < 3 (see Section A. 3). 
We now show that ipk is bounded. Since 



3pa J 1 y?logN(Bx D,d,e)de + 3, (A.9) 

it suffices to show that the right-hand side is bounded. This follows from crude upper bounds on 
the covering number. Indeed, observe that 

d((x,y),(x',y')) < \j2mR 2 max |(aj, x - x'}\ + \j2mRi max \(ai,y-y'}\ 

l<i<m' 1 l<i<m 1 1 

< \J2mR2sn \x - + \/2mR\rp \y - y'\ t ■ 
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Thus, 

n(b*d M <_n [b, Uh , i -^=) ■ n [d, Hh , i -^=) 

The second line comes from the standard volumetric estimate N(B, |-||^ ,e) < (f ) S for e < 1. The 
factor (") arises from decomposing D as the union of ( n ~ s ) sets of the same form as B, but with 
support size bounded by r. Now, in order to bound the last inequality, we further write (™) < n r 
and R\,R 2 < m. Plugging this in, we obtain 

y/log N(B x D,d,e) < C^/r + s\J\og(mn/e). 

To conclude, a simple integration gives 

J \/r + syJlog(mn/e)de < \Jr + s(y/log(mn) + 1), 

which establishes the claim since the right-hand side is dominated by cr _1 . 

A. 3 Coarse scale: k < 

This section contains the crucial estimates, which must be developed very carefully. To show that 
(fk is bounded, observe that by definition, p _feo_1 < m?l 2 , and thus -(ko + 1) < logm provided that 
log p > 3/2. It follows that ip k < 1 + (logm + l)/logm < 3. 

Next, we show that the more subtle bound (A. 7) holds. Let {(xi,yi)} be the points in the 
definition of the Majorizing measure theorem with mutual distances at least so that N = 

\\{ x ii yi)}\- Let z x be the minimizer of {||^||^ : < p~ } and let Zj be the minimizer of 

{||z||^ 2 : < p~ k }. Finally, introduce the pivotal quantity 

a— ii ii 2 ii ii 2 
(7 .- max \\Zj\\n — \\Zx\\n ■ 

l<j<N 



We must show that 

P 



k a\J\ogN < max (p k+2 (xj,yj) - p k (x,y) = + 2/logm. 



l<j<N 



In order to bound N, we consider the points {zj,yj} and note that N = \{zj,yj}\. 

We shall need two key properties of the points {zj,yj}. First, these points are well separated. 
Indeed, the triangle inequality, gives for i + j 

d((zi,yi),(zj,yj)) > d((xi,yi), (xj^j)) - d((xi,yi),(zi,yi)) - ^((xj,^), (^,%)) 
> p _fc_1 - \Fi \\xi - Zi\\ B - v2\\xj - Zj\\ B 
>p- k - 1 -2V2p- k - 2 

2 P 
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provided that p > 4\/2. Second, each Zj is close to x in the sense that 



x-Zj\\ B < \\x-Xj\\ B + \\xj-Zj\\ B < -^=d((x,y),(xj,yj)) + \\xj-Zj\\ B < -==p k + p k 2 <p 



V2 



V2 f 



provided that p 2 > 2 + v2- Therefore, it follows from the definition of z x that ||-^j||^ 2 > 



Now, the benefit of the special construction of ip^ on the coarse scale is that the size of 9 
restricts the space that {zj} can inhabit. To demonstrate this, since {z : \\z - x\\ B < p~ k } is convex, 



< p ' belongs to this set. Now combine 



2 \\e 2 



> \\z x \\ e2 with || Zj\\ fn > \\z x \\ fn to give 



III II 2 III II 2 

2 W Z 3H 2 + 2 » Zx »h 



Zj z x 



<- II II 2 - II II 2 
s W z j\\£ 2 \\ z x\\e 2 s 



Hence, 



I Zj z x || 4 < 2\/d. 



(A.10) 



Combined with Lemma A. 5, we obtain 

N < N(B(u, 2\fe) x D, d, p^ 1 / 4 ), 

where B(u, 2y9) ■- {x : supp(x) c T, \\x\\« < 1, \\x - u\\» < 2\/6}. 

Set e = p~ fe_1 /4. We cover B{u,2\f§) x D to precision e in the metric d by covering B(u,2\J~0) 
to precision e/2 under the norm \/2||-|| B and D to precision e/2 under the norm \f2mR\ \-\ D - We 
have 



\/log iV < \/log N(B (u, 2^6) , n/2 (I • || ~ , e/2) + \/log N(D, ^2mR 1 \\ ■ \\ ~ , e/2) . (A.ll) 

To bound the second term, observe that D c s/rBi x , where Be 1 is the unit ball under the l\ norm. 

Hence, 

N(D,y/2mR 1 1| • || D , e/2) < N(B h , \\-\\ D ,Ce/y/rR im ) 

and Lemma 3.7 in [35] bounds the right-hand side by following an argument from [17]. This lemma 
gives 3 



fMfrt fT^rTu /^AW^ r~ V^gl log n log m 
ylog{N(D,^2mRi \\-\\ D ,e/2)) < Csjm . 

Now we bound N(B(u, 2^0), V2 \\-\\ B , e/2) = iV(B, |-| B , e/iV29) as follows: 
^/logJV(.B,|H| B ,e/4\/20) < C—yJmsp{l + R 2 ). 



(A.12) 



(A.13) 



We postpone the proof and show how it implies (A. 7). With e = r k /4, (A.ll) together with the 
bounds (A.13) and (A.12) give 



\/logiV < Cp k y / m(log nyJprRi + yJps(l + R 2 )0} . 



Now plug in 2\/6 < 6y / \og m + l/v'logm, along with the definition of a, to give 



logm 



We do not reproduce the proof here, but encourage interested parties to read the clever and short argument. 
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as desired, thus concluding the proof of Theorem 6.2. 

Proof [(A. 13)] Using the dual Sudakov minorization (Lemma A. 6), we have 



I fa 

yJlogN(B,\\-\\ B ,e/4V26)<C^ x Esup^^zt} 2 ^) 2 . (A.14) 

e \| y^D i= i 

Since (a itT ,z T } ~ J\f(0, IKr|||,), E(a iiT ,z T ) 2 = K,t|||- We write 

m 



Esup E( a *,T, zt) (ai,y) =Esup 

yeD i=1 R 

< Esup 
R 

■=Io + h- 



i=l 

III 



Y,(( a i,T,ZT) 2 ~ || %t \\ £2 )a>i,R a*i,R 

i=l 



+ sup 
R 



E IKt||| a i,R a i,R 
i=l 



The supremum is over R obeying \R\ < s and RnT = 0. Since Io^tI^, ^ s /^> we have 

= sfj,mR2- 



h < sfisup 

R 



E a hR a i,R 

1=1 



(A.15) 



Further, since E(aj 5 T, zt) 2 - \\<H,t\\i = 0, we can use symmetrization as before to obtain 



J < 2 E sup 

R 



i=l 



where {^} is a Rademacher sequence. We use a lemma — which is a direct consequence of Lemma 
3.6 in [35] — to bound this quantity. 

Lemma A. 7 ([35]) Let vi,---,v m , m <n, be vectors in R n obeying < fiQ for each i. Let 

be a Rademacher sequence. Then 



E sup 

\R\<r 



E & v i,R v i.R 
i=l 



< k sup 

\R\<r 



T, v i,R v i,R 
i=l 



where k = Cy/r log r\/\og n\/\og m < CyrJiQ log n log" 



m. 



In order to use the lemma, condition on the value of zt and set := n\zr\ x - Set G(z) 
su Pj/eD Ei2i(cii, ZT) 2 {a,i, y} 2 . Absorbing a factor of 2 into k, the lemma gives 



J <E^G(^<Ey + E^. 

It remains to bound Ek < Cr/j, log n log mE||2y|| D . Recall that 

IztIIzj : = max |(a»T, zt}\ 2 

l<i<m 



(A.16) 
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and set a := maxj fla^T ||^ ^ s/j,. It now follows from a standard concentration bound on subexpo- 
nential random variables that 

ii 2 — 2 

E H-ZtIIc < Ca logm < Csfilogm 

(this can be derived by bounding P(||zr||£) > t) for all t and integrating). 

Finally, plug this last bound into (A. 16), and combine the result with (A. 15). This gives 

EG(z) < Crsfj 2 log n log 4 m + sfimR 2 + EG(z)/2. 

Rearranging the terms together with m > Crfilognlog 4 m give 

EG(z) < Cmsfi(l + R 2 ). 

Finally, inserting this into (A. 14) gives the conclusion. ■ 
A. 4 Concentration around the mean 

We have now proved that EX < e for any e > provided that m > C t fi[slogm v r log n log 4 m] . 
This already shows that for any fixed 5 > 0, 

P(X>5) < ~ 
o 

and so taking e to be a small fraction of 8 gives a first crude bound. However, we wish to show 
that if m > C/-i(3 [slogmvr log n log 4 m] then the probability of 'failure' decreases as . This can 
be proved using a theorem of [35] which in turn is a combination of Theorem 6.17 and inequality 
(6.19) of [27]. We restate this theorem below. 

Theorem A. 8 Let Y\,---,Y m be independent symmetric random variables taking values in some 
Banach space. Assume that \\Yj \\ < R for all j and for some norm \\-\\. Then for any integer I > q, 
and any t > 0, the random variable 



Z:~- 

obeys 



Y.YJ 

3=1 



ICY 

\P(Z> 8qEZ + 2R£ + t) < \^ — J +2exp 



t 2 



256g(EZ) 2 

In our setup, we work with a norm on positive semidefinite matrices given by 

||M|| := sup v*Mv, 
veV 

where V is given by (A.l). The rest of the details of the proof of concentration around the mean 
follows exactly as in the steps of [35, pages 11-12] and so we do not repeat them, but encourage 
the interested reader to check [35]. This is the final step in proving Theorem 2.7. 
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B Stochastic Incoherence 



In Sections 2-4, we have assumed that the coherence bound holds deterministically, and it is 
now time to prove our more general statement; that is to say, we need to extend the proof to 
the case where it holds stochastically. We propose a simple strategy: condition on the (likely) 
event that each row has 'small' entries, as to recreate the case of deterministic coherence (on this 
event). Outside of this event, we give no guarantees, but this is of little consequence because we 
will require the event to hold with probability at least 1 - 1/n. A difficulty arises because the 
conditional distribution of the rows no longer obeys the isotropy condition (although the rows are 
still independent). Fortunately, this conditional distribution obeys a near isotropy condition, and 
all of our results can be reproved using this condition instead. In particular, all of our theorems 
follow (with adjustments to the absolute constants involved) from the following two conditions on 
the distribution of the rows: 

||E aa* - I\\ < 1/(8a/tc) (near isotropy) 

maxi< t < n |a[i]|L < fi (deterministic coherence) . 

We first illustrate how to use near isotropy to prove our results. There are several results that need 
to be reproved, but they are all adjusted using the same principle, so to save space we just prove 
that a slight variation on Lemma 2.1 still holds when requiring near isotropy, and leave the rest of 
the analogous calculations to the interested reader. 

Set W ■- Eaa* and let Wt,t be the restriction of W to rows and columns in T. We first show 
that ^ 

P(\\A* t A t -W t ,t\\ ><5) <2sexp( — — 6 — r ). (B.2) 



H(s + 1) 2 + 25/3 , 

To prove this bound, we use the matrix Bernstein inequality of Section 2.1, and also follow the 
framework of the calculations of Section 2.1. Thus, we skim the steps. To begin, decompose 
A* t At - Wt,t as follows: 

m m 

m{A^A T - W t ,t) = Y,(. a k,T a t,T ~ W t ,t) ■= Y, x k- 

k=l k=l 

We have EX k = and \\X k \\ < \\a k ^ T al T - l\ + \I - Wt,t\\ ^ sfi + < (s + := B. Also, the total 
variance obeys 

||EX fe || 2 < |E(a fcir <4 iT ) 2 | < sfi\\Ea kiT a* kjT \\ = s/j,\\Wt,t\\ ^ S K 1 + ^) ^ ( s + 

Thus, a 2 < m(s + l)fi, and (B.2) follows from the matrix Bernstein inequality. 
Now, it follows from \\W t ,t - 1\\ < \\W - 1\\ < that 

P(\\A* T A T - I\\ > 7^ + 5) <2sexp( — 5 , \. 
V" ) /^(s + 1) 2 + 25/3) 

In the course of the proofs of Theorems 1.1 and 1.2 we require H^^t-JH < 1/2 for noiseless 
results and ||j4^^4t _ ^|| ^ 1/4 for noisy results. This can be achieved under the near isotropy 
condition by increasing the required number of measurements by a tiny bit. In fact, when proving 
the analogous version of Lemma 2.1, one could weaken the near isotropy condition and instead 
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require ||E aa* — I\\ < 1/8, for example. However, in extending some of the other calculations to 
work with the near isometry condition — such as (3.10) — the factor of >/n (or at least \fs) in 
the denominator appears necessary; this seems to be an artifact of the method of proof, namely, 
the golfing scheme. It is our conjecture that all of our results could be established with the weaker 
requirement || E aa* - I\\ < e for some fixed positive constant e. 

We now describe the details concerning the conditioning on rows having small entries. Fix the 
coherence bound [i and let 

E k = I max \a k [t]\ 2 < fi\ and G = n 1<k<m E k . 
[i<t<n J 

Thus G is the 'good' event (G is for good) on which maxi< t < n |a/c[i]| 2 < \i for all k. By the union 
bound, P(G C ) < mP(E^). We wish for P(G C ) to be bounded by 1/n, and so we require \i to be 
large enough so that P(-Ef) < (mn) 4 , 

Next we describe how conditioning on the event G induces the near isometry condition. Because 
of the independence of the rows of A, we may just consider the conditional distribution of a\ given 
E\. Drop the subindex for simplicity and write 

/= E[oo*] = E[aa*l£] + E[aa*l E c] = E[aa*|£] P(E) + E[aa*l E c]. 

Thus, 

||E[aa*|E] - /|| • P(E) = ||(1 - P(E))I- E[aa*l B c]|| < P(E C ) + ||E[aa*l £ c]|| . (B.3) 

We now bound ||E[oa*ls c ]||- By Jensen's inequality (which is a crude, but still fruitful, bound 
here) , 

||E[aa*l B =]|| < E[Iaa*l B c|] = E[||a||? l B c]. (B.4) 



and, therefore, 



\Uaa*\E] - I\\ < i _ ^ (P(E C ) + E[||a||| 1^]) 



Combine this with the requirement that P(^ c ) < (mn) to give 
as long as m^/n > 20. It now follows that in order to ensure near isotropy, it is sufficient that 

E[Hili S c]< 



U2 ^ J 20^E' 

It may be helpful to note a simple way to bound the left-hand side above. If /(£) is such that 

P(max|a[i]| 2 >t] < f(t), 
\l<t<n / 

then a straightforward calculation shows that 

E[\\a\\il E c]<n»f(fj,)+n f°° f(t)dt. 
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